Interacting with a Pet Robot using Hand Gestures

Pet robots are autonomous robots capable of exhibiting ~imal-like behaviors, including emotional ones. As they become more integrated into our lives, we will need a more natural way of communicating with them. Similarly, to perceive our intentions more effectively, they will need to understand human gestures. This work focuses on the real-time, visual interpretation of 2D dynamic hand gestures in complex environments. Our goal is to enable humans to communicate and interact with Yuppy, a pet robot being developed at the MIT AI Lab. The gesture lexicon consists of a set of 2D gesture classes (primitives) that include linear (vertical, horizontal, and diagonal) as well as circular (clockwise and counterclockwise) gestures. As the user makes the gesture, images taken by the robot’s camera are processed on a frame by frame basis. We assume that the user wears long-sleeved clothes and the robot is static while observing the gesture. Our strategy for interpreting hand gestures consists of: hand segmentation, feature extraction, and gesture recognition. Currently, our algorithms run in a user-specified window of attention which excludes the user’s face. We use both motion and color information to segment the hand from the cluttered background. Motion is detected by taking the image difference of three sequential RGB images. The skin-colored regions are computed by comparing the HLS representation of an input image with an a priori model of skin color in HS space. The results of these two modules are combined and the hand is chosen to be the skin-colored region with the largest area and the greatest number of displaced pixels. The motion of the hand’s centroid is tracked in real-time as new image frames are processed. Our system assumes that once the hand’s velocity exceeds a certain threshold, the user has started a gesture. As the hand moves, the horizontal and vertical displacements (dx, dy) of the hand’s centroid are stored in a feature vector until the hand pauses for 2-3 seconds. To recognize a gesture, we analyze the feature vector. For linear gestures, the (dx, dy) displacements cluster around fixed axes in the dx-dy plane: vertical gestures around the dy axis, horizontal gestures around the dx